An Upper Bound on Normalized Maximum Likelihood Codes for Gaussian Mixture Models

This paper shows that the normalized maximum likelihood (NML) code-length calculated in [1] is an upper bound on the NML code-length strictly calculated for the Gaussian Mixture Model. We call an upper bound on the NML code-length as uNML (upper bound on NML). When we use this uNML code-length, we have to change the scale of data sequence to satisfy the restricted domain. However, in the point of model selection, we find the fact that the model selection algorithm is essentially universal regardless of scale conversion of data in Gaussian Mixture Models, and the experimental results of paper [1] can be used as it is. In addition to this, we correct the normalized maximum likelihood (NML) code-length for generalized logistic distribution calculated in [1]. 1 Problem Setting In this paper, we consider the problem of model selection in which we aims to calculate the number of clusters for Gaussian Mixture Model. Let us use the given sequence x = (x1, · · · , xn), xi = (xi1, · · · , xim)> (i = 1, · · · , n). Here, we use the Gaussian Model Class: N(μ, Σ), μ ∈ R, Σ ∈ Rm×m, and calculate NML code-length for Gaussian Model. The Gaussian distribution for data sequence x is defined as follows: f (x; μ, Σ) = 1 (2π)mn 2 · |Σ | n2 exp { − 1 2 n ∑ i=1 (xi − μ)Σ(xi − μ) } . We define the NML distribution fNML relative to a model class M = { f (X; θ) : θ ∈ Θ} by fNML(x;M) def = f (x; μ̂(x), Σ̂(x)) ∫ Y f (yn; μ̂(yn), Σ̂(yn))dyn . (1) Here Y is the restricted domain for x. By using this restrict, we can calculate the NML code-length without divergence. The NML code-length for Gaussian Mixture Model is defined as below with latent variable z: LNML(x, z;Y,M(K)) def = − log fNML(x, z) = − log f (x, z;M(K), θ̂(x, z)) + log C(M(K), n), (2) C(M(K), n) = ∑ wn ∫ Y f (y,w;M(K), θ̂(y,w))dy. (3) Here θ = (π, μ, Σ) is the set of parameters, We consider problem of model selection for Gaussian Mixture Model with using (2) as a criterion. ∗S. Hirai and K. Yamanishi are with Graduate School of Information Science and Technology, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, JAPAN E-mail: so [email protected] E-mail: [email protected] 1 ar X iv :1 70 9. 00 92 5v 1 [ cs .I T ] 4 S ep 2 01 7 2 Influence of scale conversion of data on model selection When we use the NML code-length defined by (2), we have to change the scale of data sequence to safisfy the restricted domain Y (e.g. to multiply 1/α, etc.). In this section, we show that the model selection algorithm is essentially universal regardless of scale conversion of data. Let us consider the NML code-length for Gaussian Mixture Model as LNML(x, z;Y,M(K)), and we can get the definition of code-length as (2). The term influenced by the scale of the data is the first term of the formula (2). Here, in order to evaluate the influence of the first term from the scale conversion of the data, we use the dataset x α (= x/α) which we calculate by multiplting x by 1/α. We consider model selection when each data x or x α is used, and evaluate the difference between them. Since it is important for model selection to evaluate the difference between M(K1) and M(K2), we focus on the difference in the first term of equation (2). − log f (x α, z;M(K1), θ̂(x α, z)) − (− log f (x α, z;M(K2), θ̂(x α, z)) = C + K1 ∑